perf(gemma4): regular decode steps stop rebuilding what did not change - #1000

Description

Closes #999

The token read parked the caller on the stream every step and paid the OS scheduler's wake-up for it. It now spins on cuStreamQuery for a bounded window first, falling back to the parking wait so a stuck stream still sleeps.

Greedy decode rounds run a depth-two software pipeline on the one stream: the staged argmax writes the picks into the id buffer the next step's embedding reads, the readback lands in one of two pinned slots, and the emitted stream lags compute by one step. Any composition change, non-greedy row, logprob request or row within two tokens of its length cap drains to the collect-every-step path; a stop found late marks the row stopping and the drain retires it. The kernels and their order do not change, and the emitted tokens are the same bytes.

A regular step re-uploaded every per-step metadata table although their next values were already known. A kernel captured at the decode graph's tail now advances them in place, and a step whose fingerprint matches the previous one — every length one token further, page, chunk and split structure untouched — skips the rebuild and every upload. Mixed steps, chunk boundaries, page turns and the precapture warm pass invalidate the fingerprint and rebuild as before.

The staged sampler chain — suppression, argmax and the device copy of the picks — is captured per bucket at startup and launched as one graph beside the decode replay; only the pinned readback stays outside, so the collector keeps the copy's own event.

Test Env

Single GPU (sm_89, 48 GiB, x86_64)

Verification

• cargo fmt --all -- --check and git diff --check against main pass. The Gemma 4 release server builds, all-target Gemma/kernels/sample Clippy passes with -D warnings, and release lib tests pass: core 33, Gemma 42 with 20 ignored, kernels 16 with 1 ignored, sample 3.

• The engine and serving gates pass through scripts/gemma4_gates.sh, each running 1 test … 1 passed in its own process, the dual-checkpoint ones on both the 12B and the 26B: the shared and green lane lifecycles, the gathered lifecycle and the gathered walk against the serial path, the mixed step against the serial path, the overlapped prefill against the sync step, the ragged-batch row-order gate, and the 12B HF waypoint and greedy gates.

• Byte identity at pinned composition: the same four greedy public /v1/completions requests (31-, 35- and 33-token prompts plus a 1421-token one; 64 forced tokens; finish_reason=length, completion_tokens=64) answer with one md5 on main's binary and on this head's, 219de1e3c09c.

• Serving A/B, this head against main, vllm bench serve random 1024 in / 128 out at 8 / 24 / 64 prompts, two cooled alternating rounds, same seed per round, mean [range]:

  	c1	c4	c16
  output tok/s, main	106.0 [105.2–106.7]	277.4 [275.6–279.1]	594.3 [590.7–597.8]
  output tok/s, this branch	107.8 [107.6–107.9]	282.5 [278.4–286.6]	600.6 [598.7–602.4]
  median TPOT ms, main	8.57 [8.57–8.57]	12.64 [12.60–12.68]	23.77 [23.71–23.83]
  median TPOT ms, this branch	8.45 [8.44–8.46]	12.26 [12.24–12.28]	23.33 [23.19–23.47]

  Median ITL follows TPOT (c1 8.51 → 8.38, c4 11.61 → 11.19, c16 16.31 → 16.07 ms). First-token time is not a claim of this change; one c4 round read a higher p99 on this branch (586.8 against 399.9 ms) that the other round did not (401.1 against 401.6 ms).

• Node-level Nsight traces of the same request set (64 prompts at c16, then 8 at c1) on each binary: host-to-device copies 62,374 → 36,998 over the trace (the regular steps' metadata uploads gone; the bytes are the same 17.67 GB because the prefills' KV writes dominate), kernel launches 1,125,022 → 1,126,542 (one tail kernel and one sampler graph per step in place of the host-launched chain), inter-kernel gaps longer than 50 µs 1,695 → 641, summing 45.6 → 38.6 s over a trace span of 68.2 → 61.2 s with the GPU-busy total unchanged (22.49 → 22.51 s).